DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation

With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic search-free models with DyLoRA at least 4 to 7 times (depending to the task) faster than LoRA without significantly compromising performance. Moreover, our models can perform consistently well on a much larger range of ranks compared to LoRA.Comment: Accepted to EACL 202

Analysis of Cantilever Triple-Layer Piezoelectric Harvester (CTLPH): Non-Resonance Applications

In this research, a design guideline for a kinetic energy converter using a cantilever tri-ple-layer piezoelectric harvester (CTLPH) for low-frequency applications is presented. By combin-ing the constitutive and internal energy equations, the analytical equations for harvested voltage and power were developed. It was also found that frequency of motion, applied tip force, piezoe-lectric coefficients, geometrical dimensions, and mechanical properties of layers play significant roles in the performance of the harvester. Having characterised the voltage regulator module, LTC3588, the dependency of output voltage on both the storage and output capacitors of the LTC3588 was investigated. An experimental measurement using the optical method was carried out to determine the applied tip force. Furthermore, the performance of the CTLPH in low frequencies (< 3.3 Hz) for various resistive loads was investigated. It was found that both excitation frequency and external resistance load are effective on the maximum generated power. The developed CTLPH shows the optimum power of 17.3